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APPARATUS AND METHOD OF PACKETIZING DATA STREAM 



BACKGROUN D OF THE TNYFNTION 

1 . Field of the Invention 

The present invention relates generally to an apparatus and a 
method for packetizing a coded bitstream of digital data in accordance 
with a transport protocol. More particularly, it relates to an apparatus 
and a method for packetizing the bitstreams wherein time information is 
generated in accordance with a specification of the protocol, a header 
including the time information is generated, and a packet including the 
time information in accordance with the protocol is generated with 
adding the header to every predetermined unit of the bitstreams. 

2. Description of Related Art 

Many researches for digital broadcasting utilizing an MPEG4 
stream defined by the Moving Picture Experts Group, a working group of 
ISO, as ISO/IEC 14496 are going on to find an alternative to an 
MPEG2-stre«n (ISO/IEC 13818), which has been conventionally used 
for the digital broadcasting. 

Syntax of a stream of MPEG4 Visual Object Sequence defined as 
ISO/IEC 14496-2 Is shown in Figs. 1 and 2. At the top of the hierarchy as 
shown in Fig. 1 is the Visual Object Sequence. The Visual Object 
Sequence includes mostly Profile and Level indications of the stream. 
Each Visual Object included in the Visual Object Sequence describes a 
type of objects that are multiplexed on the stream. Thus, this allows the 
MPEG4 to define a plurality of visual objects such as still-frame video 



image objects, face objects, and video objects. 

The case that the video objects ate multiplexed on the stream will 
be described. Information, "visual_objectJype (4); 0x03 " described in 
the Visual Object indicates that the type of Visual Object is video object. 
If the type of Visual Object is video object and the Video Objects are 
multiplexed, the Visual Object includes Video Object Layer- The Video 
Object Layer includes a type of Video Object, a bit rate of the stream, a 
size of a buffer of the receiver side, occupancy of data to be stored in the 
buffer of the receiver side up to a start of the decoding, and a frame rate. 

Herein, the information of 30 bits, "first Jha!fJ>itjrate (15)" and 
u latter_half_vbv_bit_rate (15)", described in the Video Object Layer 
indicates the bit rate of the stream. The information of 18 bits, 
"first^half^vbv^buffer^sizc (15)* and «IatterJialf_vbv Jmffer^size (3) 5? 7 
described in the Video Object Layer indicates a size of the buffer of 
receiver side. The information of 22 bits, "firstJkalf_vbv_occupancy 
(11)" and "latterJ*alf_vbv_occupancy (11)", described in the Video 
Object Layer indicates the occupancy of the data to be stored in the 
receiver side up to a start of the decoding. The information, 
u vop_time_increment_resoIution (16)" and "fixed_vop_time w increi2ient 
(1-16)", contains information to get the frame rate. Although the 
information from "first haif_bit_rate (15)" to 
"latter_baJf_ybv_occupancy (1 1)" as described above exists when 
"vbvjparameters (1)" are set to 0x1, it does not exist when they are set 
to 0x0 where t4 0x" indicates hexadecimal digit 

The Video Object Layer may include Group of Video Object Plane. 



fails to multiplex time information such as elementary stream clock 
reference (hereinafter called ESCR), decoding time stamp (hereinafter 
caHed DTS), presentation time stamp (hereinafter called PTS) 9 which is 
necessary for the synchronization in a packet. Thus, it is necessary to 
transport the MPEG4 Visual Object Sequence with adding the time 
information in order to attain the synchronization* 

Japanese Patent Publication, JP-H11-98193-A, discloses a 
translation of MPEG4 stream into M?EG2-?E$ packet. It, however, 
discloses no particular way of translation- 
It is an object of the present invention to present an apparatus and a 
method capable of obtaining a packet including the time information in 
accordance witb the transport protocol. 

SUMMARY OF THE mVKNTmN 

According to the present invention, the foregoing object is attained 
by an apparatus for packetizing coded bitstreams of digital data in 
accordance with a transport protocol. The apparatus comprises lime 
information generating unit for generating time information in 
accordance with a specification of the transport protocol, and header 
generating unit for generating a header that includes the time 
information generated by the time information generating unit- It further 
comprises packet generating unit for generating a packet with adding the 
header generated by the header generating unit into every predetermined 
unit of bitstream. 

In accordance with one aspect of the invention, a method of 



accordance with the specification of the transport protocol to be 
generated. 

Further, the time information in accordance with the specification 
of transport protocol is generated using information from an encoder for 
generating the coded bitstream of digital data. The time information is 
also generated using maximum bit rate of the coded bitstream of digital 
data. 

The apparatus further comprises a stream analyzer for analyzing the 
coded bitstream of digital data, and an arithmetic unit for performing 
arithmetic operation on, namely, calculating time until a decoder for 
decoding at least the coded bitstream of digital data stuns decoding the 
bitstream on the basis of a result of the analysis of the stream analyzer. 
According to the apparatus, the time information in accordance with the 
specifications of transport protocol is generated using an arithmetic 
result of the arithmetic unit. 

Such the generated time information in accordance with the 
specifications of transport protocol is included in the header. A packet is 
generated with adding the header to every predetermined unit of the 
bitstreams. Thus, the packet contains the time information in accordance 
with the specification of transport protocol- Therefore, a receiver 
receiving the packet containing the time information can synchronously 
reproduce the bitstreams using the time information. 

Since the packet includes the time information, receiver side carries 
out the decoding thereof without a failure and performs the 
synchronization with another stream easily. 
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In accordance with another aspect of the invention, an apparatus for 
packotizing a plurality of coded bitstreams of digital data in accordance 
with respective transport protocols is provided. The apparatus comprises 
a plurality of time information generating units each for generating the 
time information in accordance with specifications of each of the 
transport protocols. The apparatus further comprises a plurality of 
header generating units each for generating a header including a 
plurality of the pieces of time information generated by the time 
information generating units, and a plurality of packet generating units 
each for generating a packet with adding the headers generated by the 
header generating unit into every predetermined unit of the plurality of 
bitstreams. According to the apparatus, the plurality of time information 
generating units respectively generates time information in accordance 
with the specification of each of the transport protocols using common 
time information. 

In accordance with further aspect of the invention, a method of 
packezizing a plurality of coded bitstreams of digital data in accordance 
wUh respective transport protocols is provided. The method comprises 
time information generating step of generating a plurality of pieces of 
the time information in accordance with the specification of each of the 
transport protocols. The method further comprises header generating 
step of generating a plurality of headers each including the plurality of 
pieces of the time information generated during the time information 
generating step. It also comprises packet generating step of generating a 
plurality of packets with adding the headers generated during the header 



generating step into every predetermined unit of a plurality of the 
bitstreams. During the time information generating step, a plurality of 
pieces of the time information in accordance with the specification of 
each of the transport protocols is generated using common time 
information. 

According to this aspect of the invention, when packetizing a 
plurality of coded bitstreams of digital data in accordance with 
specification of each of the transport protocols, for example, translating 
a plurality of the MPEG4 streams into each of the MPEG2-PES packets, 
a plurality of pieces of the time information is first generated in 
accordance with specifications of each of the transport protocols. 

In this case, a plurality of pieces of the time information in 
accordance with specification of each of the transport protocols is 
generated using common time information. Thus, the plurality of pieces 
of the time information is generated on same time axis. 

Such the plurality of pieces of the generated time information in 
accordance with the specification of each of the transport protocols is 
inserted into each of the headers. A packet is generated with adding the 
headers to every predetermined unit of the plurality of the bitstreams, 
Thus, the packet contains the time information in accordance with the 
specifications of transport protocols. Therefore, a receiver receiving the 
packet containing such the time information can synchronously 
reproduce the bitstreams using the time information. 

Since the packet includes the time information ? receiver side carries 
out the decoding thereof without a failure and performs the 
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synchronization with another stream easily. 

In accordance with stiii another aspect of the invention, the 
apparatus further comprises storing device for storing the coded 
bitstreams every predetermined unit of the coded bitstreams. According 
to this aspect of the invention, time information generating unit 
generates the time information, which is necessary for a synchronous 
transport in accordance with the transport protocol, every the 
predetermined unit, Packet generating unit generates a packet with 
adding the time information generated by the time information 
generating unit into every the predetermined unit stored in the storing 
device. 

In accordance with still further aspect of the invention, the 
apparatus further comprises storing device of storing the MPEG4 stream 
every predetermined unit of the MPEG4 stream. According to this aspect 
of the invention, time information generating unit generates the time 
information, which is required to transport the bitsiream in accordance 
with an MPEG2-PES stream protocol, every the predetermined unit of 
MPEG4, Packet generating unit generates a packet with adding the time 
information generated by the time information generating unit into every 
the predetermined unit of the MPE04 stream stored in the storing device. 

In accordance with another aspect of the invention, the apparatus 
further comprises detecting unit of detecting from the MPEG4 stream the 
information that is required to generate the time information required 
when the MPEG2-PES stream is transported. It also comprises time 
information generating unit for generating the time informatioe every the 



predetermined unit of MPEG4 stream on the basis of the information 
detected by said detecting unit. 

The conclusion unit of this specification particularly points out and 
distinctly claims the subject matte of the present invention. Those skill 
in the art, however, will best understand both the organization and the 
method of operation of the invention, together with further advantages 
and objects thereof, by reading the remaining portion of the specification 
in view of the accompanying drawing(s) wherein like reference 
characters refer to like elements. 



BRIEF DESCRIPTION OF THE DRAWING^ 

Fig,! is a diagram depicting a structure of MPEG4 stream (part 

one); 

Fig. 2 is a diagram depicting a structure of MPEG4 stream (pan 
two); 

Fig. 3 is a block diagram depicting configuration of a transport 

system according to a first embodiment of the present invention; 
Fig. 4 is a block diagram depicting configuration of a transport 

system according to a second embodiment of the present invention; 
Fig, 5 is a block diagram depicting configuration of a transport 

system according to a third embodiment of the present invention- 
Fig. 6 is a block diagram depicting configuration of a transport 

system according to a forth embodiment of the present invention: 
Fig. 7 is a block diagram depicting configuration of a transport 

system according to a fifth embodiment of the present invention; 
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Fig. 8 is a block diagram depicting configuration of a transport 
system according to a sixth embodiment of the present invention; 

Fig. 9 is a block diagram depicting configuration of a transport 
system according to a seventh embodiment of the present invention; 

Fig. 10 is a block diagram depicting configuration of a transport 
system according to an eighth embodiment of the present invention; 

Fig, 1 1 is a diagram depicting an exemplary configuration of 
computer for packetizing data stte&m; 

Fig. 12 is a flowchart for an explanation of a flow of packetizing 
data stream; and 

Fig, 13 is a flowchart for an explanation of another flow of 
packetizing data stream. 

DETAILED PESCRTPTIOtt OF THE TOFFRRRET> 

KMBOPTMBNTtf) 

1 , Transport System 

(1) First Embodiment of the Invention 

Fig. 3 illustrates in a block diagram form configuration of a 
transport system 100A as a first embodiment of the invention The 
transport system 100A comprises encoder 101 for encoding and 
transferring MPE04 stream STM1, a packccizer 102A for receiving the 
MPEG4 stream STM1 from the encoder 101, translating it into MPEG2- 
PES packet and transferring a stream STM2 of MPEG2-PES packet form, 
and a multiplexer 103 for receiving and multiplexing the MPEG2-PES 
packet stream STM2 from the packetizer 102 A and transporting a stream 
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STM3 of MPEG2-TS form as a transport output signal, 

According to the transport system 100A ? the MPEG4 stream STM1 
transferred from tlie encoder 101 is supplied to the packetizer 102A, 
which translates it into MPEG2-PES packet. The stream STM2 of 
MPEG2-PES packet form transferred from the paeketizer 102A is 
supplied to the multiplexer 103, which transports the stream STM3 of 
MPEG2-TS form as its output signal. The multiplexer 103 sections the 
information, MPEG4 video^descriptor and multiplexes the stream STM2 
and the information, MPEG4 video_descriptor thus sectioned to indicate 
that the data contained in the stream STM2, which the multiplexer 103 
receives, is MPEG4 stream. 

The packetizer 102A comprises a header detecting unit 111 for 
receiving MPEG4 stream STM1 and detecting headers from the stream 
STM1, a header buffer unit 112 for receiving the headers detected by the 
header detecting unit 1 1 i 7 storing them every header, and detecting 
information required to generate time information to be inserted into the 
header of MPEG2-PES packet, a time information generating unit 113A 
for generating the time information such as ESCR, DTS, and PTS using 
the information detected by the header buffer unit 112 and initial value 
of ESCR 5 which is supplied from outside, 

The packetizer 102A further comprises a PES header generating unit 
1 1 4 for generating a PES header using the time information such as 
ESCR, DTS, and PTS generated by the time information generating unit 
113A and the information, streamed supplied from outside. It also 
comprises a buffer unit 115 for storing received MPEG4 stream STM1 in 
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sequence every one access unit {hereinafter called AU) and a PES* 
packetizing unit 1 16 for generating MPEG2-PES packet with adding the 
PES header generated by the PES header generating unit 114 into a head 
of the data of one AU stored in the buffer unit 115. Here, although the 
essential components of MPEG4 stream STM1 of one AU are VOPs ? a 
first AU comprises a header portion of upper layer before a first VOP* 

The following operation of the packetizer 102 A will be explained 
more in detail. The header detecting unit 111 receives MPEG4 stream 
STM1 and detects the headers from the stream STM1- 

The header detecting unit 111 counts a number of AUs assuming 
that one AU is detected if the unit 111 detects the headers more than the 
VOP headers contained in the above Group of Video Object Plane when 
detecting the VOP headers, and keeps the counts as a variable 
(hereinafter called AUcnt). The header detecting unit 111 also counts a 
number of bytes In each AU and keeps them as a variable (hereinafter 
called AU[iJ) on one AU basis where "i" is an integer included in the 
imerval 1 ^ i^ AUcnt and indicates detected order of the A Us. The time 
information generating unit 113 A uses these variables (AUcnt and 
AU[i]) when generating the time information such as ESCK, DTS, and 
PTS, as will be described hereinafter. When MPEG4 data has completed, 
it processes final AU of the MPEG4 stream STMt as if it might detect 
one AU. 

The buffer unit 115 receives input MPEG4 streams STM1 and stores 
them in sequence on one AU basis. The header buffer unit 112 receives 
The detected headers from the header detecting unit 111 and stores them 
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one by one. The header buffer unit 112 also analyzes stored headers and 

detects the time information as follows: 

fir$ijialf_bit_rate and latter_half_bit_rate each indicating bit 
rate of the MPEG4 stream STM1 are detected from a header of the 
Video_ObjectJLayer; first^half^vbv^occupancy and 
latter_ha!f_vbv_occupancy each indicating an occupancy of the data 
to be stored in the buffer unit until a receiver side starts decoding 
are detected from a header of the Video_Object_Layer; 
vop_dme_increment_resolution and fixed_vop_time_incrernent 
each obtaining frame rate are detected from a header of the 
Video_Object_Layer; vop_coding_type indicating I-picture, P- 
picture or B-picture is dQUQUd from a header of the 
Yideo^Objcct Plane; modulo_time_base indicating how rn&ny 
seconds it takes for a picture to occur after I-pictuie is detected 
from a header of the Video_Object_P2ane; and vop_time_increment 
indicating how order a picture should occur, namely, the order of 
reproduction is dotect^d from a header of the Video^ObjectjPianc. 

The time information generating unit 113A receives the above 
information detected by the header buffer unit 312, The unit 1I3A also 
generates the time information such as ESCR, £>TS, and PTS using the 
information obtained by analyzing the headers, the variables (AUcnt and 
AU[i]) stored in the header detecting unit III, and initial value of ESCR, 
which is supplied from outside, 

I will explain operations for generating the time information, The 
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unit 1 13A initially gets the information, DTSJ3ffset, indicating the time 
until the receiver side starts decoding after receiving the data. I may 
calculate the value of the information, DTS^Offset, by the data 
occupancy, vbv^occupancy, to be stored in the buffer unit and bit rates, 
bitbrace, as follows: 

If vbv_occupency * fir$t_half_ v b v _occupancy * OxSQCO 4 

!atterJialf_vhv_occnpancy and bit_rate fir;$tjialf_bit_rate * 

OxSOOO + latterjhalf J>it_me, then 

DTS_Offsex [sec] - (64 * vbv^occupancy)/(bitjrate * 400) 

where oxSOOO indicates 8000 in hexadecimal. 

Next, the unit 113 A gets the time (T) to be updated on one AU basis. 
I may calculate the value of the time (T) by the information, 
fixed_vop_time_incrcment and voptime^increment^resolution, which 
have been already detected to calculate the frame rate, as follows: 

T [sec] = 

fixed_vop_time_increment/vop_time_increment_resolntion. 

According to these calculated values, I may get decoding time, DTS ? 
for "Pth AU as follows: 

DTS - T*i + DTS_Offset + £SCRJ>ase 
where ESCR_base indicates initial value of ESCR. 

The unit 1 13A also counts a number (m_cnt) of "1" included in the 
information, modulo jrime^base, the number, m_cnt, indicating how 
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many seconds it takes for a picture to occur after I-picture already 
detected. Namely, the information, m_cnt indicates a number of *1* 
included in the information, modulo_time_ba$e- 

The unit 113 A gets the information indicating reproduction time 
<vop_time). 1 may calculate the value of the time (vop^time) by the 
information, vop_time_jmcremcnt indicating an order of reproduction 
and the above vop^time^increment^resolution as follows: 

vop_tirne *= vop_time^increment/vop_tiiiie_increment_resolutioa, 

According to these calculated values, 1 may get reproduction time, 
PTS ? for "i n th AU with putring the information, vop_time, m_cnt ? 
DTS w Offset 7 and ESCR_base together with each other as follows: 
PTS - vop^tiine + m^cnt + DTS ^Offset + ESCR_base. 

Therefore, I may also get reference time, ESCR, for "i"th AU as 
follows: 

ESCR - 1/bi^rate * 8 * ( X AU [i-l] + ESCR_byte) + 
ESCRbase 

where ESCR_byte indicates a number of bytes occurred from a top 
of the PES header to an ESCR field every AU and 2 AU fi-1] indicates a 
sum of the values from AU [1] to AU [M], 

The PES header generating unit 1 14 receives the time information, 
ESCR, DTS 5 PTS thus generated by the unit 1 13 A, The unit 1 14 
generates PES header using the time information and stream 
identification information, streamjd, supplied from outside. The unit 
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1 14 generates the PBS header on one detected AU basis- 

The unit 1 14 initially gets values in the interval between OxEO and 
OxEF of the stream identification information, streamed, from outside 
where "Ox" indicates that succeeding flgure(s) is (are) hexadecimal 
digit(s). If PTS is equal to DTS, then the unit 114 gets PTS and ESC&. If 
not equal, it gets PTS. DTS S and ESCR together with PTS_DTS_f]ags 
and ESCR_flag. It also calculates and gets PES_header_data_length 
according to added time information, 

An embodiment of the PES headers is illustrated as follows: 
packex„start_code_prefIx « 0x000001; /*24 [bit]*/ 



streamed * OxEX; /*$ [bit]*/ 

PES_packet_Iength - 0x0000; /*16 [bit]*/ 

opnonal_PES_header - 0x2; /*2 [bit]*/ 

PES_scrambling_coritrol * 0x00; /2 [bit]*/ 

PES^prioTity - 0x0; j\ bit]*/ 

data_alignment_indicator = Oxl /I bit]*/ 

copyright ■ 0x0; /I bit]*/ 

original_or_copy * 0x0; f\ bit]*/ 

PTS_DTS_fIags - OxXX; /2 bit]*/ 

ES CRjflag = 0x1; /I bit]*/ 

ES_rate_f!ag - 0x0; /I bit]*/ 

DSM_trickjaode_flag - 0x0; /] bit]*/ 

additional_copy_info_iIag » 0x0; /l bit]*/ 

PES_CRC_flag = 0x0; /I bit]*/ 

PES_extension_flag - 0x0; and 11 bit]*/ 
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PES_header_dataJength = OXxx 



/*8 [bit]*/ 



Finally, the PES -packeti zing unit 116 receives the PES header thus 
generated by the above unit 1 14, The unit 1 16 generates MPEG2-PES 
packet with adding the PES header generated by the unit 114 into a. head 
of the data of one AU stored in the buffer unit 115. 

Although this embodiment illustrates the above fixed values in the 
PES header, another value is available according to any applications. 

Due to the first embodiment of the invention, the transport system 
obtains the time information such as ESCR, DTS, and PTS from the 
information on the MPEG4 stream STM1. It generates the PES header 
with including the time information therein and generates the MPEG2- 
PES packet with adding the PES header into the MPEG4 stream every AU, 
thereby transporting MPEG2-TS stream STM3 as its output signal. Since 
PES header of the MPEG2-PES packet includes the time information 
such as ESCR. DTS, and PTS, receiver side carries out the decoding 
thereof without a failure and the indication synchronizing with another 
stream. 

(2) Second Embodiment of the Invention 

Pig, 4 illustrates in a block diagram form configuration of a 
transport system 100B as a second embodiment of the invention. Since, 
in Fig. 4 7 like reference characters refer to like elements of Fig. 3 3 
detailed explanation of these c]ements is omitted. 

The transport system 1G0B comprises encoder 101 for encoding and 
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transferring MPEG4 stream STMI, a packetizer 102B for receiving the 
MPE04 stream STMI from the encoder 102, translating it into MP£G2~ 
PES packet and transferring a stream STM2 of MPEG2-PES packet form, 
and a multiplexer 103 for receiving and multiplexing the MPEG2-PES 
packet stream STM2 from the packctizer 102B and transporting a stream 
STM3 of MPEG2^TS form as a transport output signal. 

According to the transport system 100B, the MPEG4 stream STMI 
transferred from the encoder 101 is supplied to the packetizer 102B, 
which translates it into MPEG2-PES packet* The stream STM2 of 
MPEG2-PES packet form transferred from the packetizer 102B is 
supplied to the multiplexer 103, which transports the stream STM3 of 
MPEG2-TS form as its output signal. 

The packetizer 102B receives another stream STM4 of MPEG2-PES 
packet form from outside. The packetizer 102B detects PES header from 
the stream STM4, generates the time information such as ESCR, DTS, 
and PTS using information obtained by analyzing the PES header and 
includes it In the PES header. 

The multiplexer 1 03 also receives the stream STM4 and transports it 
as the stream STM3 of the MPEG2-TS form. It also sections the 
information, MPEG4 video_descriptor and multiplexes the STM2 and the 
information, the MPEG4 video_descriptor thus sectioned to indicate that 
data contained in the STM2, which the multiplexer 103 receives, is 
MPEG4 stream. 

The packetizer 102B further comprises a header detecting unit 117 
for receiving and detecting PES headers from the stream STM4 of 
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MPEG2-PES packet form and transferring them, a header analyzing unit 
1 IS for receiving and analyzing the PES header detected by the unit i 17 
and obtaining necessary information therefrom, and a Time information 
generating unit 113B for generating the time information such as ESCR, 
DTS, and PTS using the information given by the units 1 12 and 118. 
Remaining units of the packetizer 102B like the ones of the packetizer 
102 A in the transport system 100A as shown in Fig.3 are respectively 
formed. 

The stream STM4 of MPEG2-PES packet form is supplied to the 
header detecting unit 117, which detects the PES headers. The unit 117 
detects one PES header from the stream STM4 whenever the header 
detecting unit 111 detects one header from the stream STM1 of the 
MPEG4 stream form. 

The PES headers thus detected are supplied to the header analyzing 
unit 118, which analyzes them, The unit 118 detects the information* 
reference time (ESCR'), decoding time (DTS')*tcproduction time (PTS 3 ) ? 
a number of bytes (BSCR_byte') indicating a number of bytes occurred 
up to an ESCR' field, and ES_rate\ 

The time information generating unit 113B receives the above 
information detected by the header analyzing unit 118. The unit 113B 
gets the time informaticn such as PTS, DTS, and, ESCR, for^T'th AU as 
follows: 

PTS - vop^time + m^cnt + PTS ; 
DTS = T*I + DTS';and 

ESCR - l/bit_rate*8*( E AU [i-ij + ESCR_byce) + £SCR_base 
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where ESCRJ>ase - ESCRVsystem^clock^frequency + (0- 
ESCRJjyte'^ES^rate^SO), the above information, 
system_clock_frequency t is standardised according to MPEG2 standard, 
namely, 90 kHz, and remaining vales of the above formulae are 
respectively the same as those used in the first embodiment, 

The PES header generating unit 114 receives the time information 
such as PTS, DTS, and, ESCR, thus generated by the time information 
generating unit 113B. It also generates other PES headers using the time 
information and the information, streamed, supplied from outside. Since 
remaining operations of the packetizer 102B are the same as those of the 
packetizer 102 A of the transport system 100 A as shown in Fig. 3, 
detailed explanation thereof is omitted. 

According to the second embodiment, the transport system 100B 
obtains the time information such as ESCR, DTS, and PTS from the 
information on MPEG4 stream STM1 and the information on another 
stream STM4 of the MPEG2-PES packet form- It generates the PES 
headers with including the time information therein and generates the 
MPEG2-PES packet with adding the PES header into the MPEG4 stream 
every AU/ thereby transporting MPEG2-TS stream STM3 as its output 
signal. Since PES header of the MPEG2-PES packet includes the time 
information such as ESCR, DTS, and FTS, receiver side carries out the 
decoding thereof without a failure and the indication synchronizing with 
another stream. 

Further, since, according to the second embodiment the transport 
system 100B obtains the time information such as ESCR, DTS, and PTS 
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from the information on another stream STM4 of the MPEG2-PES packet 
form, the input MPEG4 stream is translated into the stream STM2 of 
MPEG2-PES packet form with the stream STM4 being synchronized with 
the stream STM2, Therefore, receiver side may reproduce synchronously 
the stream STM2 with the stream STM4. 

(3) Third embodiment of the Invention 

Fig. 5 illustrates in a block diagram form configuration of a 
transport system 10OC as a third embodiment of the invention. Since, in 
Fig, 5, like reference characters refer to like elements of Fig- 3, detailed 
explanation of these elements is omitted. 

The transport system 100C comprises encoder 101 for encoding and 
transferring MPEG4 stream STM1, a packetizer 102C for receiving the 
MPEG4 stream STMl from the encoder 101, translating it into MPEG2- 
PES packet and transferring a stream STM2 of MPEG2-PES packet form, 
and a multiplexer 103 for receiving and multiplexing the MPEG2-PES 
packet stream STM2 from the packetizer 1G2C and transporting a stream 
STM3 of MPEG2-TS form as a transport output signal, 

According to the transport system 1Q0C, the MPEG4 stream STMl 
transferred from the encoder 101 is supplied to the packetizer 102C, 
which translates it into MPEG2-PES packet. The stream STM2 of 
MPEG2-PES packet form transferred from the packetizer 102C is 
supplied to the multiplexer 103, which transports the stream STM3 of 
MPEG2-TS form as its output signal. 

The packetizer 102C receives information, vbv_occupancy, 
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bytes, of accessed AU and gets the information, bit w rate, indicating a bit 
rate of the stream STM1, 

The time information generating unit 113C receives the information, 
bitjrate, thus obtained by the emit J 19. The unit 1 13C also receives the 
information detected by the header buffer unit 112, It receives the 
information, vbv occupancy, and initial value of ESCR from outside. 

The unit 113C generates the time information such as ESCR, DTS, 
and PTS using the information, vbv_occupancy and bit_rate, and the 
information detected by the header buffer unit 1 12 as the necessary 
information other than the above one, like the time information 
generating unit 1 13 A of the transport system 100A as shown in Fig.3. 

The PES header generating unit 114 receives the time information, 
ESCR, DTS, PTS thus generated by the unit I 13C. The unit 14 generates 
the PES header using the time information and stream identification 
information, streamed, supplied from outside- Since remaining 
configuration and operations of the packetizcr 102C are the same as 
those of the packetizcr 102 A of the transport system 100 A as shown in 
Fig. 3 5 detailed explanation thereof is omitted. 

According to the third embodiment, the transport system 100C 
obtains the time information such as ESCR> DTS, and PTS from the 
information on the MPEG4 stream STM1, the information, bit^rate, 
calculated by the bit rate calculating unit 119 and the information, 
vbv_occupancy, supplied from outside. It generates the PES header with 
including the time information therein and generates the MPEG2-PES 
packet with adding the FES header into the MPEG4 stream every AU, 
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thereby transporting MPEG2-TS stream STM3 as its output signal. Since 
the PES header of the MPEG2-PES packet includes the time information 
such as ESCR, DTS, and PTS, receiver side carries out the decoding 
thereof without a failure and the indication synchronizing with another 
stream. 

Further, according to the third embodiment, the transport system 
100C uses the information, biwate, calculated by the bit rate calculating 
unit 119 and the information, vbv_occupancy, supplied from outside 
when it obtains the time information such as ESCR, DTS, and PTS, Thus, 
if there is no information for getting the information, bitjrate and 
vbv occupancy on the stream STM1, it appropriately generates the time 
information. 

(4) Forth embodiment of the Invention 

Fig. 6 illustrates in a block diagram form configuration of a 
transport system 100D as a forth embodiment of the invention. Since, in 
Fig, 6, like reference characters refer to like elements of Fig. 3 ? detailed 
explanation of these elements is omitted. 

The transport system 100D comprises encoder 101 for encoding and 
transferring MPEG4 stream STM1, a packetizer 102D for receiving the 
MPEG4 stream STM1 from the encoder 101, translating it into MPEG2- 
PES packet and transferring a stream STM2 of MPEG2-PES packet form, 
and a multiplexer 103 for receiving and multiplexing the MPEG2-PES 
packet stream STM2 from the packetizer 102D and transporting a stream 
STM3 of MPEG2-TS form as a transport output signal. 
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According to the transport system 100D 7 the MPBG4 stream STM1 
transferred from the encoder 101 is supplied to the paeketizer 102D, 
which Translates it into MPEG2-PES packet. The stream STM2 of 
MPEG2-PES packet form transferred from the packetizer I02D is 
supplied to the multiplexer 103 f which transports the stream STM3 of 
MPEG2-TS form as its output signal. 

The packetizer 102D receives the information, vbvoccupancy, 
indicating an occupancy of the data to be stored in the buffer unit until 
receiver side starts decoding, and the information, bitjrate, indicating a 
bit rate of the MPEG4 stream STM1, respectively from the encoder 201. 
The packetizer I02D generates the time information such as ESCR ? DTS, 
and PTS using the information, vbvoccupancy and bit_rate. The 
packetizer 102D also includes the time information into the PES header 

The multiplexer 103 receives and sections the information, MPEG4 
video_descriptor and multiplexes the STM2 and the MPEG4 
video_descriptor thus sectioned to indicate that data contained in the 
STM2 ? which the multiplexer 103 receives, is MPEG4 stream. 

The time information generating unit 11 3D receives the information^ 
vbvoccupancy and bit_rate ? from the encoderlOl. It also receives the 
information detected by the header buffer unit 112 and initial value of 
ESCR from outside. 

The unit 11 3D generates the time information such as ESCR, DTS, 
and PTS using the information, vbv^occupancy and bit_rate, supplied 
from the encoder 101 and the information detected by the header buffer 
unit 1 12 as the necessary information other than the above onel like the 
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time information generating uait 113 A of the transport system 100A as 
shown in Fig.3, 

The PES header generating unit 114 receives the rinse information, 
ESCR, DTS, PTS thus generated by the unit II 3D. The unit 1 14 
generates the PES header using the time information and stream 
identification information, streamed, supplied from outside. Since 
remaining configuration and operations of the packetizer 102D are the 
same as those of the packetizer 102A of the transport system 10OA as 
shown in Fig. 3, detailed explanation thereof is omitted. 

According to the forth embodiment, the transport system 100D 
obtains the time information such as ESCR, DTS„ and PTS from the 
information on the MPEG4 stream STM1, and the information, 
vbv_occupancy and bit_rate ? supplied from the encoder 1 01. It generates 
the PES header with including the time information therein and generates 
the MPEG2-PES packet with adding the PES header into the MPEG4 
stream every AU, thereby transporting MPEG2-TS stream STM3 as its . 
output signal. Since the PES header of the MPEG2-PES packet includes 
the time information such as ESCR, DTS, and PTS r receiver side carries 
out the decoding thereof without a failure and the indication 
synchronizing with another stream. 

Further, according to the forth embodiment, the transport system 
100D uses the information, vbvoccupancy an<3 tartrate, supplied from 
the encoder 101 when it obtains the time information such as ESCR, DTS, 
and PTS. Thus, if there is no information for getting these bitjrate and 
vbvoccupancy on the stream STM1, it appropriately generates the time 
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information. 

(5) Fifth embodiment of the Invention 

Fig- 7 illustrates in a block diagram form configuration of a 
tra&sport system 100E as a fifth embodiment of the invention. Since, in 
Fig. 7, like reference characters refer to like elements of Fig. 3, detailed 
explanation of these elements is omitted. 

The transport system I00E comprises encoder 101a for encoding 
and transferring MPEG4 stream STM1a, apacketizer 102Ea for receiving 
the MPEG4 stream STMla from the encoder 101a, translating it into 
MPEG2-PES packet and transferring a stream $TM2a of MPEG2-FES 
packet form, encoder 101b for encoding and transferring MPEG4 stream 
STMlb, a packetizer i02Eb for receiving the MPEG4 stream STMlfe 
from the encoder 101b, translating it into MPEG2-PES packet and 
transferring a stream STM2b of MPEG2-PES packet form, and a 
multiplexer 103 for receiving and multiplexing the MPEG2-PES packet 
streams STM2a and STM2B from the packetizers ]02Ea and 102Eb and 
transporting a stream STM3 of MPEG2-TS form as a transport output 
signal. 

According to the transport system I0CE, the MPEG4 stream STMJa 
transferred from the encoder 101a is supplied to the packetizer 202£a, 
which translates it into MPEG2-PES packet. The MPEG4 stream STMtb 
transferred from the encoder 101b is supplied to the packetizer 102Eb, 
which translates it into MPEG2-PES packet. The streams STM2a and 
STM2b of MPEG2-PES packet form transferred from the packetizers 
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102Ea and 102£b are supplied to the multiplexer 103, which transports 
the stream STM3 of MPEG2-TS form as its output signal. 

The multiplexer 103 receives and sections the information, MPEG4 
video^descriptor and multiplexes the streams STM2a and STM2b and the 
MPEG4 Wdeo_dcscnptor thus sectioned to indicate that data contained 
in the multiplexed MPEG2-PES packet is MPEG4 stream. 

Each of the packetizers, 102£a and 102Eb, is $o configured as to be 
the same as the packetizer 102A of the transport system 100 A as shown 
in Fig. 3. The time information generating unit 1 13A of the packetizer 
102Ea generates the time information such as ESCR, DTS 3 and PTS using 
the information detected by the header buffer unit 112 and initial value 
of ESCR supplied from outside. The PES header generating unit 1 14 of 
the packetizer 102Ea generates the PES header with including the time 
information therein. 

The time information generating unit 113A of the packetizer 102£h 
generates the time information such as ESCR, DTS, and PTS using the 
information detected by the header buffer unit 112 and initial value of 
ESCR supplied from outside. The PES header generating unit 114 of the 
packetizer i02Eb generates the PES header with including such the time 
information therein. The initial value supplied to the time information 
generating unit 11 3 A of the packetizer l02Eb is set to be the same as the 
one supplied to the time information generating unit 1 13 A of the 
packetizer 202Ea. 

Since remaining operations of the packetizers 102Ea and 102Eb are 
the same as those of the packetizer 102A of the transport system 1 00A as 
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shown in Fig, 3> detailed explanation thereof is omitted. 

According to the fifth embodiment, the transport system 100E 
obtains the time information such as ESCR, DTS, and PTS from the 
information on the MPEG4 streams STMla and STMlb. It generates the 
PES header with including the time information therein and generates the 
MPEG2-PES packet with adding the PES header into the MPEG4 stream 
every All, thereby transporting MPEG2-TS stream STM3 as its output 
signal. Since PES header of the MPEG2-PES packet includes the time 
information such as ESCR> DTS, and PTS, receiver side carries out the 
decoding thereof without a failure and the indication synchronizing with 
another stream- 
Further, according to the fifth embodiment, each of the time 
information generating units 113A, 113A of the packetizcrs I02Ba and 
102Eb receives common initial value of ESCR and generates the time 
information such as ESCR, DTS, and PTS, thereby generating the time 
information based on the same time axis. Therefore, the packetizers, 
102Ea, 102Eb, sequentially transfer MPEG2-PES packets each having 
the PES header including the time information generated according to 
the same time axis, 

(6) Sixth embodiment of the Invention 

Fig. 8 illustrates in a block diagram form configuration of a 
transport system 100F as a sixth embodiment of the invention. Since, in 
Fig. 8, like reference characters refer to like elements of Fig. 3 ? detailed 
explanation of these elements is omitted. 
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The transport system 100F comprises encoder 101 for encoding and 
transferring MPEG4 stream STM1, a packetizer 102F for receiving the 
MPEG4 stream STMt from the encoder 101, translating it into MPEG2- 
PES packet and transferring a stream STM2 of MPEG2-PES packet form, 
and a multiplexer 103 for receiving and multiplexing the MPEG2-PES 
packet streams STM2 from the packetizer 102F and transporting a stream 
STM3 of MPEG2-TS form as a transport output signal- 
According to the transport system 100F, the MPEG4 stream STM1 
transferred from the encoder 101 is supplied to the packetizer 102F, 
which translates it into MPEG2-PES packet. The stream STM2 of 
MPEG2-PES packet form transferred from the packetizer 102F is 
supplied to the multiplexer 303, which transports the stream STM3 of 
MPEG2-TS form as its output signal. The multiplexer 103 receives and 
sections the information, M?EG4 vjdeo^descriptor and multiplexes the 
stream STM2 and the MPEG4 video_descriptor thus sectioned to indicate 
that the data contained in the multiplexed MPEG2-PES packet is MPEG4 
stream. 

The packetizer 102F comprises a header detecting unit 111 for 
receiving MPEG4 stream STM1 and detecting headers from the stream 
STM1, a header buffer unit 112 for receiving the headers detected by the 
header detecting unit 1 1 1, storing them one by one, and detecting 
information required to generate the time information such as ESCR, 
DTS, and PTS to be inserted into the header of MPEG2-PES packet, and 
a time information generating unit 113F for generating the time 
information using the information detected by the header buffer unit 1 12 ? 



31 



initial value of ESCR supplied from outside, and maximum bit rate 
AURmax supplied from outside. 

The packetizer 102F further comprises a PES header generating unit 
114 for generating a PES header using the time information such as 
ESCR, DTS, and PTS generated by the time information generating unit 
113F and the information, streamed, supplied from outside. 

It also comprises a buffer unit 115 for storing input MPEG4 stream 
STM1 in sequence on one AU basis, and a PES-packetizing unit 1 16 for 
generating MPEG2-PES packet with adding the PES header generated by 
the PES header generating unit 114 into a head of the data of one AU 
stored in the buffer unit 1 15, Here, the essential components of MPEG4 
stream STM1 of one AU are VOPs, and a first AU comprises a header 
portion of upper layer before a first VOP, 

In the packetizer 102F, the MPEG4 stream STM1 transferred from 
the encoder 101 is supplied to the header detecting unit 111, which 
detects the headers on the stream. The headers detected by the unit 111 
are supplied to the header buffer unit 112, which stores them one by one. 

The header detecting unit 1 1 1 counts a number of AUs assuming 
that one AU is detected if the unit 1 1 1 detects the header other than the 
VOP header when detecting the VOP header, and keeps the counis as a 
variable (AUcnt), Further it also counts a number of byres in each AU 
and keeps it as a variable (AU[i]) every AU where rt i" is a number 
included in the interval, 1 g iS AUcnt and indicates detected order of the 
AUs. The time information generating unit 1 13F uses these variables 
(AUcnt and AU[i]) when generating the time information such as ESCR, 
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DTS ? and PTS, as will be described hereinafter. When MPEG4 data has 
completed, it processes final All of the MPEG4 stream STM1 as if it 
might detect one AtL 

The buffer unit 115 receives input MPEG4 streams STM1 and stores 
them in sequence on one AU basis. The header buffer 112 analyzes the 
headers stored therein and detects the information as follows: 

fir$tJiaIf_vbv_occupancy, latter Jialf_vbv_occupancy, 

first Jkalf_bit_rate, latter_half_bit_raxe, 

vop_time_increment_resoIution, and fixcd_vop_time_increment, 
are detected from a header of the Video_ObjectJLayer; and 

vop_coding_iype, modulojtimejbase, vop^timejncrement, are 
detected from a header of the VideojDbject_Layer> 

The time information generating unit 2 13F receives the above 
information detected by the header buffer unit 112. The unit 113F 
generates the time information such as ESCR, DTS, and PTS using the 
information obtained by analyzing the headers, the variables (AUcnt and 
AU[ij) kept in the header detecting unit 1 1 1 ? maximum bit rate, AURmax, 
supplied from outside and Initial value of the ESCR supplied from 
outside. 

I will explain operations for generating the time information. The 
unit I13F initially calculates an occupancy of the data, vbv_occupancy, 
to be stored in the buffer of receiver side until starting the decode afteT 
receiver side receives the data as follows: 

vbv_occupancy « firstJialf_vbv_occupancy * 0x8000 + 
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latter_half_vbv_occupancy. 



Next, the unit 113F gets the time (T) to be updated on one AU basis 
by calculating it as follows: 
T [sec] - 

fixcd^vop^time^increcnent/vop^time^iijcrcinent^rcsolution. 

According to the following operations, the unit 1 13F gets DTS, PTS 
and ESCR, each for T'th AU, where I set differential time when a 
preceding AU is PES-packetized (the exce$3 time over time for one 
frame) as S (mi [sec]- 

1- Time AUT [i] for transporting «Pth AU (size AU [ij) is 
calculated as follows: 

AUT [i] = T~S^ 

2. Transporting rate AUR fi] when transporting "i"th AU (size AU 
[i]) Is calculated as follows: 

If AUT [i] ^ 0, then AUR [i] » AURmax; and 

If AUT [i] > 0 f then AUR [i] = AU [i]/AUT [i] 

3 . According to the calculated AUR [i], time for real transport 
AUT [i] is calculated as follows: 

AUT [i] = AU [i] / AUR [i] 

4. The differential time Si to be used when "i+l^th AU is PES- 
packettzed is calculated as follows: 

Si - S w ■+ AUT [I] - T 

5, According to the calculated bit rate AUR [i], DTSJDffset is 
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calculated as follows: 

DTS_Offeet = (64*vbv_occupancy)/ AUR [i] 

6. According to the results thereof, DTS (decoding time) is 
calculated as follows: 

DTS » T*i + DTS jDffset + ESCR^basc 

where ESCR_base <* initial value of ESCR 

7. A number (m_cnt) of "l" included in the information, 
modulo_tune_base is counted. The number, m_cnt, indicates how many 
seconds it takes for a picture to occur after I-picture, 

Therefore, 

indent « a number of "1" Included in modulo jtime^base 

K. The information, vop_time, for indicating the reproduction time 

is calculated as follows: 

vop_time ~ vop_time_increment/vop_time_xncrement^resoiution 
9> According to these calculated values, reproduction time, PTS, is 

calculated as follows: 

PTS - vop_time + indent + DTS^Offset + ESCR_base 

10. A number of bytes from a top of the PES header to an ESCR 
field is calculated on one AU basis and they are set as ESCR_byte, 

1 1. According to the results thereof, reference time, BSCR 4 is 
calculated as follows: 

ESCR - 1/AUR [i] * 8 * (2 AU + ESCRJ>ytc) + 
ESCRJ>ase 

The PES header generating unit 114 receives The time information. 



ESCR, DTS, PTS thug generated by the time information generating unit 
1 13P. The unit 14 generates the PES header using these time information 
and stream identification information^ stream_id f supplied from outside. 
The unit 1 14 generates the PES header on one AU basis, 

The PES-packeiizing unit 1 16 receives the PES headers thus 
generated by the above unit 114, The unit 116 generates MPEG2-PES 
packet with adding the FES header generated by the unit 114 into a head 
of the data of one AU stored in the buffer unit 115. 

According to the sixth embodiment, the transport system 10GF 
obtains the time information such as ESCR, DTS, and PTS from the 
information on the MPEG4 streams STM1 and the information of 
AURmax supplied from outside, It generates the PES header with 
including the time information therein and generates the MPEG2-PES 
packet with adding the PES header into the MPEG4 stream every All, 
thereby transporting MPEG2-TS stream STM3 as its output signal. Since 
PES header of the MPEG2-PES packet includes the time information 
such as ESCR, DTS, and PTS ? receiver side carries out the decoding 
thereof without a failure and the indication synchronizing with another 
stream. 

Further, according to the sixth embodiment, the time information 
generating unit 1 1 3F receives the information, AURmax, indicating 
maximum bit rate from outside, thereby generating the time information 
with limiting maximum value of the bit rate. Therefore, this allows the 
PES-packetizing to be performed at any bit rates that are available to a 
transport line. 
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(7) Seventh embodiment of the Invention 

Fig. 9 illustrates in a block diagram form configuration of a 
transport system 100G as a seventh embodiment of the invention. Since, 
in Fig. 9, like reference characters refer to like elements of Fig, 3, 
detailed explanation of these elements is omitted. 

The transport system 100O comprises encoder 101 for encoding and 
transferring MPEG4 stream STM1, a stream analyzer 3 04 for receiving 
and analyzing the MPEG4 stream STM1 from the encoder 101 and 
detecting predetermined information, and an arithmetic unit 105 for 
performing arithmetic progression on the frame rate FR, the bit rate AUR 
ft] every AU, the transport time AUT [i] every AU, and the reproduction 
lime DTS [i] every AU according to the information detected by the 
stream analyzer 104, The arithmetic unit 105 receives maximum bit rate 
AURmax, threshold value of Offset, receiving buffer size Bmax, and 
initial vahje of ESCR. 

The transport system 100G further comprises a packetizer 102G for 
receiving MPEG4 stream STM1 through the stream analyzer 104 7 
translating it into MPEG2-PES packet and transferring a stream STM2 of 
MPEG2-PES packet form, and a multiplexer 105 for receiving and 
multiplexing the MPEG2-PBS packet streams STM2 from the packctizer 
i02G and transporting a stream STM3 of MPEG2-TS form as a transport 
output signal. 

According to the transport system 100G 5 the MPEG4 stream STM1 
transferred from the encoder 10 Ms supplied to the stream analyzer 104. 
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The analyzer 104 analyzes the stream STMT to detect the following 
information; 

Total number of AUs in a streamy AUcnt; 
Size of each of the AUs, AU fij, 

where *T' is an integer included in the interval between I ^ 
AUcnl and indicates 4i i"th AU from the top thereof (so does the 
following): 
vopjtime_incremeat_resolution; and 
fixed_vop_time_increment 

The stream STM1 thus analyzed in the analyzer 104 is supplied to 
the packetizcr 102G, which translates it into MPEG2-PES packet. 

The arithmetic unit 105 receives the information thus detected by 
the analyzer 104. The unit 105 calculates the information such as the 
frame rate FR 5 the bit rate AUR [i] every AU r the transport time AUT [i] 
every AU, and the reproduction time DTS {ij every AU according to the 
information detected by the analyzer 104. 

The respective pieces of the above information will be calculated as 
follows: 

(a) Frame Rate 
FR = 

vop_tirne_increment_re&olution/fixed_vop_tirne_incrernerit; 

(b) Bit Rate every AU and Transport Time every AU 
Suppose that differential time after transferring AU [i-i] is 
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a S". The differential time S is set to the integral of the 
difference between the transport time if AU is transported at 
1/FR, which means time for representing data of one frame, 
and real transport time AUT [ij. If the differential time S is not 
less than 1/FR, then AUR [i] is set to maximum bit rate 
AURmax. This AURmax is supplied from outside. Therefore, 

SS l/FR -* AUR [i] = AURmax 
If the differential time S is less than 1/FR ? then the formula, 
AUR [i] = AU [i]/ (1/FR-S), is used and AUR [i] is calculated. 
Therefore, 

S<1/FR ~» AUR [i] « AU [J]/ (1/FR-S) 
If calculated AUR [i] exceeds AURmax, then it is modified to 
AURmax. 
Therefore, 

AUR[i]> AURmax -» AUR [i] » AURmax 
According to the fixed AUR [i], a formula, AUT [i] - AU [ij / 
AUR [13, is used and the transport time AUT [i] of Tth AU is 
calculated on the oasis of the formula* 
Therefore, 

AUT [i] - AU [i]/AUR [i] 
The differential time S is updated following a formula, S *- S + 
AUT [i] « (1/FR) 
;) Reproduction Time every AU 

The differential, DTS^Offset, between the initial value of 
the ESCR and DTS is firstly set to one {sec]. Thereafter, the 
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differential, DTS^Offset, is calculated and modulated. As the 
initial value of the ESCR, the one supplied from outside is 
used. 

Therefore, 

DTS [i] - (l/FR)*i + DTS_Offset 4- ESCRJ?ase 

The arithmetic unit 105 performs advanced calculation on variations 
in the data occupancy in the buffer being ready for a receiver side to 
prevent the receiver side from a failure. Generating the time information 
according to the calculated results reliably allows it to be translated into 
PES without a failure of receiver side. 

According to the following operations, the arithmetic unit 105 gets 
the data occupancy, B_occ [i], stored in the buffer of the receiver side 
until the time of DTS p] on one AU basis. 

1- The time between DTS and DTS fi] is set to SJ, it is 
calculated as follows: 
SJ-DTS [i] and DTS 

2. Regarding a first AU, however, SJ is calculated as follows: 
SJ = DTS [lj- ESCR J>ase 

3, When preceding AU is decoded, if AU [i'l] is being input, the 
buffer receives the remaining data SD J. if a first AU, SD 

is set to zero. 
Therefore, 

SJ>SD[i-l]/AUR0-l] ~> B_occ[ij~ 
B^occ[i] + SD 
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SJ = SJ-SD[i.l]/AUR[Hl 

4. New input data of AU, for example, data from "j"th AO to 
halfway of "j+2 n th AU> is input. Whenever the buffer receives 
the data, it subtracts the time, for example, AUTft], for 
receiving the data from SJ aad then stops receiving the data 
when SJ falls to zero. The data of "j 13 th AU is supplied to the 
buffer. 

Therefore, 

SJf>AUT[j3^B^occ 

B_occIiJ+AUR0]*AUT03 
SJ=SJ-AUT0] 
The data of "j+l"th AU is supplied to the buffer. 
Therefore, 

SJ>AUT(j+l]~*B_occ [i]= 
B_occfi]+AUS0+l]«AUT[j+l] 
SJ=SJ^AUTU+13 
The. data of **j+2"th AU is supplied to the buffer. 
If AU[j*2] exceeds the SJ 7 then the data of the SJ is supplied. 
Therefore, 

SJ £ AUT0+2j~>B_occ [i]=B_occ[i]+AUR[j^2rSi 

5. Remaining data, SD (J J, of AU[j+2] is calculated as follows: 
SD[q-AUS0+2]-AUR[j+2]*SJ 

6. A number, of finally received AU is stored, 

7. The initial value of data occupancy of the buffer after decoding 
the data of M i"th AU is calculated. The data of *i+l w th AU is set 
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10 the calculated initial value of the data occupancy of buffer 
until decoding the data of "i+Pth AU. 
Therefore, 
B,occ[i+]]= B_occp]-AUri} 
S. Next, the process goes to step.l in order to get data occupancy 
of the buffer until decoding the data of w i+l"th AU. 

The above operations are repeated up to decoding of the data of all 
the AUs so that the data occupancy, B__occ of the buffer when 
respectively decoding the data of the AUs can be obtained. Tt detects "i* 
having maximum number of calculated occupancy, B_occ [i], and gets 
the differential BS between buffer size Bxnax received from outside and 
the B_occ [i]. 

Therefore, 

BS-Braax - B_occ r i3 

If BS^ SD[i], then, it divides BS by bite rate AURD+2] of the data 
of AU last received when calculating the occupancy, B_occ[i] and gets 
the time, Offset, for transferring the data of BS, 
Therefore, 

If Offset-BS/AURO+2] and BS>S,D[i] s then the Offset is calculated 
as follows: 

Offsct=SD{i]/AUR{j+2] + (BS-SD[!}/AURD+33) 

According to the Offset value thus calculated, the information, 
DTS_Offset, is re-calculated. Then DTS[i] is again gotten on an AU 
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basis. If Offset value exceeds tfee interval between Offset^ a 
threshold value of the Offset, then it re-calculates variations of the data 
occupancy of the buffer on receiver side and newly gets values of the 
Offset. 

The time information generating unit 113G of the packetizer 102G 
receives the above information, DTS[i] and DTS_Offset every AU, frame 
rate FR, bit rate AUR[i] every AU, and the initial value of ESCR, each 
calculated by the arithmetic 105, The unit H3G generates the time 
information such as ESCR and PTS using the information supplied from 
the arithmetic unit 105 and the information detected by the header buffer 
unit 112, like the time information generating unit 113F of the transport 
system 100F, 

The PES header generating unit 114 receives the time information, 
ESCR, DTS and PTS from the time information generating unit 3 13G* 
The unit 14 generates the PES header using the above rime information 
and stream identification information, streamed, supplied from outside. 
The unit 114 generates the PES header on detected one AU basis. 

The PES-packetizing unit 116 receives the PES header thus 
generated by the above PES header generating unit 114. The unit 116 
generates MPEG2-PES packet with adding the PES header generated by 
the unit 114 into a head of the data of oae AU stored in the buffer unit 
115. 

The stream STM2 of MPEG2-PES packet form transferred from the 
packetizer 102G is supplied to the multiplexer 103, which transports the 
stream STM3 of MPEG2-TS form as its output signal. The multiplexer 

43 



103 sections the information, MPEG4 video_descriptor and multiplexes 
the stream STM2 and the MPEG4 video_descriptor thus sectioned to 
Indicate that data contained in the STM2, which the multiplexer 103 
receives, is MPEG4 stream. 

Since remaining constitution and operations of the transport system 
100G are the same as those of the transport system 100A as shown in Fig- 
3, detailed explanation thereof is omitted. 

According to the seventh embodiment, the transport system 1G0G 
obtains the time information such as ESCR, DTS, and FTS from the 
information on the MFEG4 streams STMl and the information supplied 
from the arithmetic unit 105. It generates the PES header with including 
the time information therein and generates the MPEG2-PES packet with 
adding the PES header into the MPEG4 stream every All, thereby 
transporting MPEG2-TS stream STM3 as its output signal. Since PES 
header of the MPEG2-PES packet includes the time information such as 
ESCR, DT3> and PTS, receiver side carries out the decoding thereof 
without a failure and the indication synchronizing with another stream. 

Further, according to the seventh embodiment, the arithmetic unit 
105 calculates the information, DTS_Offset without using the 
information, vbvoccupancy, on the data occupancy until the start of 
decoding in the buffer of receiver side, thereby generating the time 
information such as DTS and PTS in the arithmetic unit 105 and the time 
information generating unit H3G using the DTS^Offset, Therefore, this 
allows the transport system 100G to generate the time information 
suitably if the information, vbv_occupancy> is unavailable* 
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(8) Eighth Embodiment of the Invention 

Fig. 10 illustrates in a bJoclc diagram form configuration of a 
transport system 100H as an eighth embodiment of the invention. Since, 
in Fig. 10, like reference characters refer to like elements of Fig, 3, 
detailed explanation of these elements is omitted. 

The transport system 100H comprises encoder 101 for encoding and 
transferring MPEG4 stream STM1, a packetizer 102H for receiving the 
MPEG4 stream STM1 from the encoder 101, translating it into MPEG2- 
PES packet and transferring a stream STM2 of MPE02-PES packet form, 
and a multiplexer 103 for receiving and multiplexing the MPEG2-PES 
packet streams STM2 from the packetizer 102H and transporting a 
stream STM3 of MPEG2-TS form as a transport output signal. 

The packetizer 102H comprises a stream analyzer 104 for receiving 
and analyzing the MPEG4 stream STM1 from the encoder 101 and 
detecting predetermined information, and an arithmetic unit 105 for 
performing arithmetic progression on the frame rate FR ? the bit rate AUR 
[i] every AU> the transport time AUT [i] every AU, and the reproduction 
time DTS [i] every AU according to the information detected hy the 
stream analyzer 104. The arithmetic unit 105 receives maximum bit rate 
AURmax, a threshold value of Offset^ receiving buffer size Brnax, and 
initial value of ESCR from outside. 

The packetizer 102H further comprises a header detecting unit 1 ] 1 
for receiving MPEG4 stream STM1 and detecting headers from the STM1, 
a header buffer unit 112 for receiving the headers detected by the header 
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detecting unit 111, storing them one by one, and detecting information 
required to generate the time information to be included in a header of 
MPEG2-PES packet, a time information generating unit U3H fox 
generating the time information such as ESCR, DTS, and PTS using the 
information detected by the header buffer unit 112 and the information 
supplied from the arithmetic unit 105. 

The packetizer I02H also comprises a PES header generating unit 
1 14 for generating a PES header using time information such as ESCR, 
DTS, and PTS generated by the time information generating unit 1 13H 
and the information, streanajd, supplied from outside. It also comprises 
a buffer unit 115 for storing received MPEG4 stream STM1 in sequence 
on one AU basis and a PES-packetizing unit 1 16 for generating MPEG2- 
PES packet with adding the PES header generated by the PES header 
generating unit 114 into a head of the data of one AU stored in the buffer 
unit 115. 

The transport system 1Q0H shown in Fig. 10 is the same as the 
system 10GG shown in Fig. 9 except that the packetizer 102H includes 
the stream analyzing unit 104 and the arithmetic unit 105, while the 
packetizer 100G excludes them. This allows the system 100H to operate 
like the system 100G. The system 100H obtains the same effect as the 
system 100G. 

2, Packetiziag Processes 

I will illustratively explain packetizing processes, which the 
packetizer, for example, packetizer 102A of the transport system 100A 
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as shown in Fig. 3 performs, with a computer 150 as shown in Fig. 11 
implementing them. 

The computer 150 comprises CPU 151, read only memory 
(hereinafter called ROM) 152 for storing operation program of the CPU 
151 etc, ^ random access memory (hereinafter called RAM) 153, and 
storing medium such as hard disk drive {hereinafter called HDD) 154. 
All these units, CPU 151, ROM 152, RAM153, and HDD 154 are 
connected with bus 155. The HDD 154 stores MPEG4 streams STM1 
before the translation and streams STM2 of the MPEG2-PES packet form, 
obtained according to the packetizing processes, 

(1) Packetizing Process relating to Packetizer 102A of Transport 
System 100 A embodying the Invention, shown in Fig, 3. 

Such the packetizing process will be explained referring to a 
flowchart as shown in Fig, 12. 

The process starts at Step 1. The transport system determines 
whether the MPEG4 data, which the HDD 154 stores as the input data, is 
complete at Step 2, If not complete, then it reads new MPEG4 data as 
input data from the HDD 154 at Step 3. 

It determines whether the read MPEG4 data includes a header at 
Step 4. If not included, then the buffer in the RAM 153 stores the data at 
Step 5 and the process goes to Step 2. 

If included, the header buffer in the RAM 153 stores data of this 
header at Step 6 and it determines whether data of one AU is detected at 
Step 7. In this case, if a header exceeding next VOP header is detected 
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when it detects the VOP header, then it is set as the data of one AU, Thus, 
as the data of one AU is detected, the buffer in the RAM 153 stores the 
MPEG4 data of one AU. 

If the data of one AU is not detected at Step 7, then the buffer in the 
RAM 153 stores the MPEG4 data read at the Step 5 and the process goes 
to Step 2, 

If the data of one AU is detected at Step 1, then it analyzes the data 
of the header stored in the header buffer in the RAM 153 and detects the 
information,, first_half_vbv_occupancy, latter_half_vbv_occupancy, 
first_haIfjMt_rate, latter JhalfJ>itjrate, vop_time w increment_resoIution ? 
and fixed_vop_time_increment 7 from Video_ObjectJL.ayer header and 
the information, vop_coding_type, moduio^time^base, and 
vop_time_incrcfficist' ff from Video_Object_Plane header, as described in 
the above description of transport system 100A shown in Fig. 3, at Step 8, 

It generates the time information such as ESCR, DTS and PTS, at 
Step 9, using the information detected at the Step 8 and the initial value 
of ESCR previously stored in the RAM 153. It then generates the FES 
header with including. the time information therein at Step 30, 

It generates MPEG2^PES packet, at Seep 11, with adding the PES 
header generated at the Step 10 into the MPE<34 data of one AU stored in 
the buffer of the RAM 153 and writes it into the HDD 154 as the output 
data_ 

The process goes to the Step 2 and shifts to a new MPEG2-PES 
packet generating process. Repeating the processes allows the input 
MPEG4 data to be translated into the MPEG2-PES packet in sequence 
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every AU. 

Alternatively, if the MFEG4 data as the input data is already 
complete at the Step 2, the process shifts to processing of the MPE04 
date of final AU stored in the buffer of the RAM 153. According to the 
processing, it analyzes the data of header stored in the header buffeT of 
the RAM 153 and detects the same information as the one detected at the 
StepS, at Step 12, 

It generates the time information such as ESCR, DTS and PTS, at 
Step 13, using the information detected at the Step 12 and the initial 
value of ESCR, It then generates the PES header with including the time 
information therein at Step 14. 

It also generates final MPEG2-PES packet, at Step I5 y with adding 
the PES header generated at the Step 14 into the MPEG4 data of final one 
AU stored in the buffer of the RAM 153 and writes it into the HDD 154 
as the output data. The process finishes at Step 1$. 

(2) Facketizing Process relating to Packetizer 102B of Transport 
System 1Q0B embodying the Invention shown in Fig, 4, 

Such the packetizing process will be explained referring to a 
flowchart as shown in Fig. 13. In Fig, |3 5 like step numbers refer to like 
step numbers of Fig. 12. 

The process starts at Step ] and it goes to Step 2 or 22. The 
processes starting from Steps 2 and 22 are carried out in parallel. 

Since the process starting from the Step 2, namely, Steps 2 through 
16 is the same as the one described in a flowchart as shown in Fig, 12 
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except for Steps 9 and 13, only these Steps 9 and 13 will be explained, 

Although it generates the time information such as ESCR, DTS and 
PIS, at Steps 9 and 13 in Fig-12, using the information detected by 
analyzing the header data in MPEG4 data and the information, 
ESCR_base, indicating the initial value of ESCR previously stored in the 
RAM 153, it generates the time information such as ESCR, DTS and PTS, 
at the Steps 9 and 13 in Fig- 13, using the information on reference time 
(ESCR 5 ), decoding time <DTS') ? reproduction time (PTS 1 ), a number of 
bytes until ESCR' field (ESCRJiyte'), and ES_rate' each detected by 
analyzing the PES header of MPEG2-PES packet and the information 
detected by analysing the header data in MPE04 data, at Steps 22 or 
later, as described above in the description of the transport system 100B 
shown in Fig. 4. 

The process starting from Step 22 will be explained. The transport 
system reads streams of the MPEG2-PES packet form other than the ones 
generated by translating the MPEG4 data thereinto from the HDD 154 at 
Step 22. It determines whether the read MPEG2 data includes PES header 
at Step 23. If not included, then the process goes to the Step 22. The 
Steps 22 and 23 are repeated until it finds that the read MPEG2 data 
includes PES header. 

If included at the Step 23, then the header buffer of the RAM 1 53 
stores the data of the PES header at Step 24. The data of the PES header 
stored in the header buffer of the RAM 153 is analyzed and the above 
information such as ESCR\ DTS\ PTS\ ESCR_byte\ and ESjrate% is 
detected at Step 25, 
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It then determines whether the information detected at the Step 25 
has been used in the time information generating step, Step 9 or 13, at 
Step 26. If used 3 the process goes to the Step 22 and shifts to a new 
process for detecting the information from next PES header included in 
the MPEQ2 data* which the HDD 154 reads out. This allows the process 
starting from the Step 22 to detect one PBS header of the MPEG2 
whenever the process starting from the Step 2 detects the MPBG4 stream 
of one AU. 

Thus, the packetizing process according to the flowchart as shown 
in Fig. 13 allows the input MPEG4 data to be translated into MPEG2- 
PES packet in sequence on an AU basis like the process according to the 
flowchart as shown in Fig, 12. 

The computer 150 also carries out the packetizing processes relating 
to packetizers 102C through 102 H of the transport systems 100C 
through 100H embodying the invention, as shown in Figs, 3 through 8, 
depending on their situation and cases, The detailed explanation thereof, 
however, is omitted. 

In the above embodiments, the MPEG4 stream (the MPEG4 data) is 
translated into the MPEG2-PES packet whereas the invention may also 
apply to another apparatus and another method of packetizing a coded 
bttstream of digital data in accordance with its transport protocol. 

While tbe foregoing specification has described preferred 
embodiments of the present invention, one skilled in the art may make 
many modifications to the preferred embodiments without departing 
from the invention in its broader aspects. The appended claims therefore 
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are intended to cover all such modifications as fail within the true scope 
and spirit of the invention. 



m 



m 



52 



9 



